Image selection apparatus, image selection method, and non-transitory computer-readable medium

ABSTRACT

A query acquisition unit (610) acquires pose information (hereinafter, described as query pose information) that is information to be a query and indicates a pose of a person. A search information acquisition unit (620) acquires a plurality of pieces of search pose information. The search pose information is pose information about an image (hereinafter, described as a target image) to be a search target, and is stored in a database by a plurality of target images. A selection unit (630) selects, from among the plurality of pieces of search pose information, two or more pieces of the search pose information whose degree of similarity to the query pose information satisfies a reference. The selection processing becomes substantially processing of selecting a target image similar to an image (hereinafter, described as a query image) associated with the query pose information.

TECHNICAL FIELD

The present invention relates to an image selection apparatus, an imageselection method, and a program.

BACKGROUND ART

In recent years, in a surveillance system and the like, a technique fordetecting and searching for a state such as a pose and behavior of aperson from an image of a surveillance camera is used. For example,Patent Documents 1 and 2 have been known as related techniques. PatentDocument 1 discloses a technique for searching for a similar pose of aperson, based on a key joint of a head, a hand, a foot, and the like ofthe person included in a depth video. Patent Document 2 discloses atechnique for searching for a similar image by using pose informationsuch as a tilt provided to an image, which is not related to a pose of aperson. Note that, in addition, Non-Patent Document 1 has been known asa technique related to a skeleton estimation of a person.

Meanwhile, when an image similar to an image to be a query is searched,a plurality of images are often included in a search result. PatentDocument 3 describes that images are displayed in decreasing order of adegree of similarity to an image to be a query.

RELATED DOCUMENT Patent Document

-   Patent Document 1: Japanese Patent Application Publication    (Translation of PCT Application) No. 2014-522035-   Patent Document 2: Japanese Patent Application Publication No.-   Patent Document 3: Japanese Patent Application Publication No.    2012-003623

Non-Patent Document

-   Non-Patent Document 1: Zhe Cao, Tomas Simon, Shih-En Wei, Yaser    Sheikh, “Realtime Multi-Person 2D Pose Estimation using Part    Affinity Fields”, The IEEE Conference on Computer Vision and Pattern    Recognition (CVPR), 2017, P. 7291-7299

SUMMARY OF THE INVENTION Technical Problem

The present inventor has considered that a degree of reliability of apose estimation result of an image to be a search target influencesappropriateness of a search result when an image including a person issearched with reference to a pose of the person. One of challenges to besolved by the present invention is to make it easier for a user torecognize an influence of a degree of reliability of a pose estimationresult of an image to be a search target on appropriateness of a searchresult when an image including a person is searched with reference to apose of the person.

Solution to Problem

The present invention provides an image selection apparatus including:

a query acquisition unit that acquires query pose information that isinformation to be a query and indicates a pose of a person;

a search information acquisition unit that acquires a plurality ofpieces of search pose information that are information generated foreach of a plurality of target images to be a search target and indicatea pose of a person included in the target image;

a selection unit that selects, from among the plurality of pieces ofsearch pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and

a display control unit that displays, on a display unit, the targetimage associated with each of the two or more pieces of the search poseinformation selected by the selection unit, and also sets a displayposition of the target image by using a degree of reliability of thesearch pose information.

The present invention provides an image selection method including,

by a computer:

acquiring query pose information that is information to be a query andindicates a pose of a person;

acquiring a plurality of pieces of search pose information that areinformation generated for each of a plurality of target images to be asearch target and indicate a pose of a person included in the targetimage;

selecting, from among the plurality of pieces of search poseinformation, two or more pieces of the search pose information whosedegree of similarity to the query pose information satisfies areference; and

displaying, on a display unit, the target image associated with each ofthe two or more pieces of the search pose information selected by theselection unit, and also setting a display position of the target imageby using a degree of reliability of the search pose information.

The present invention provides a program causing a computer to include:

a query acquisition function of acquiring query pose information that isinformation to be a query and indicates a pose of a person;

a search information acquisition function of acquiring a plurality ofpieces of search pose information that are information generated foreach of a plurality of target images to be a search target and indicatea pose of a person included in the target image;

a selection function of selecting, from among the plurality of pieces ofsearch pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and

a display control function of displaying, on a display unit, the targetimage associated with each of the two or more pieces of the search poseinformation selected by the selection unit, and also setting a displayposition of the target image by using a degree of reliability of thesearch pose information.

Advantageous Effects of Invention

According to the present invention, an influence of a degree ofreliability of a pose estimation result of an image to be a searchtarget on appropriateness of a search result is easier to recognize fora user when an image including an object is searched with reference to apose of the person.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-described object, the other objects, features, and advantageswill become more apparent from suitable example embodiments describedbelow and the following accompanying drawings.

FIG. 1 is a configuration diagram illustrating an outline of an imageprocessing apparatus according to an example embodiment.

FIG. 2 is a configuration diagram illustrating a configuration of animage processing apparatus according to an example embodiment 1.

FIG. 3 is a flowchart illustrating an image processing method accordingto the example embodiment 1.

FIG. 4 is a flowchart illustrating a classification method according tothe example embodiment 1.

FIG. 5 is a flowchart illustrating a search method according to theexample embodiment 1.

FIG. 6 is a diagram illustrating a detection example of skeletonstructures according to the example embodiment 1.

FIG. 7 is a diagram illustrating a human model according to the exampleembodiment 1.

FIG. 8 is a diagram illustrating a detection example of the skeletonstructure according to the example embodiment 1.

FIG. 9 is a diagram illustrating a detection example of the skeletonstructure according to the example embodiment 1.

FIG. 10 is a diagram illustrating a detection example of the skeletonstructure according to the example embodiment 1.

FIG. 11 is a graph illustrating a specific example of the classificationmethod according to the example embodiment 1.

FIG. 12 is a diagram illustrating a display example of a classificationresult according to the example embodiment 1.

FIG. 13 is a diagram for describing the search method according to theexample embodiment 1.

FIG. 14 is a diagram for describing the search method according to theexample embodiment 1.

FIG. 15 is a diagram for describing the search method according to theexample embodiment 1.

FIG. 16 is a diagram for describing the search method according to theexample embodiment 1.

FIG. 17 is a diagram illustrating a display example of a search resultaccording to the example embodiment 1.

FIG. 18 is a configuration diagram illustrating a configuration of animage processing apparatus according to an example embodiment 2.

FIG. 19 is a flowchart illustrating an image processing method accordingto the example embodiment 2.

FIG. 20 is a flowchart illustrating a specific example 1 of a heightpixel count computation method according to the example embodiment 2.

FIG. 21 is a flowchart illustrating a specific example 2 of the heightpixel count computation method according to the example embodiment 2.

FIG. 22 is a flowchart illustrating the specific example 2 of the heightpixel count computation method according to the example embodiment 2.

FIG. 23 is a flowchart illustrating a normalization method according tothe example embodiment 2.

FIG. 24 is a diagram illustrating a human model according to the exampleembodiment 2.

FIG. 25 is a diagram illustrating a detection example of a skeletonstructure according to the example embodiment 2.

FIG. 26 is a diagram illustrating a detection example of a skeletonstructure according to the example embodiment 2.

FIG. 27 is a diagram illustrating a detection example of a skeletonstructure according to the example embodiment 2.

FIG. 28 is a diagram illustrating a human model according to the exampleembodiment 2.

FIG. 29 is a diagram illustrating a detection example of a skeletonstructure according to the example embodiment 2.

FIG. 30 is a histogram for describing the height pixel count computationmethod according to the example embodiment 2.

FIG. 31 is a diagram illustrating a detection example of a skeletonstructure according to the example embodiment 2.

FIG. 32 is a diagram illustrating a three-dimensional human modelaccording to the example embodiment 2.

FIG. 33 is a diagram for describing the height pixel count computationmethod according to the example embodiment 2.

FIG. 34 is a diagram for describing the height pixel count computationmethod according to the example embodiment 2.

FIG. 35 is a diagram for describing the height pixel count computationmethod according to the example embodiment 2.

FIG. 36 is a diagram for describing the normalization method accordingto the example embodiment 2.

FIG. 37 is a diagram for describing the normalization method accordingto the example embodiment 2.

FIG. 38 is a diagram for describing the normalization method accordingto the example embodiment 2.

FIG. 39 is a diagram illustrating a hardware configuration example ofthe image processing apparatus.

FIG. 40 is a diagram illustrating one example of a functionalconfiguration of a search unit 105 according to a search method 6.

FIG. 41 is a flowchart illustrating one example of processing performedby the search unit 105 according to the search method 6.

FIG. 42 is a diagram illustrating one example of a method fordetermining a display position of a target image being performed in stepS340.

FIG. 43 is a flowchart illustrating a modification example 1 of FIG. 41.

FIG. 44 is a flowchart illustrating a modification example 2 of FIG. 41.

DESCRIPTION OF EMBODIMENTS

Hereinafter, example embodiments of the present invention will bedescribed with reference to the drawings. Note that, in all of thedrawings, a similar component has a similar reference sign, anddescription thereof will not be appropriately repeated.

(Consideration for Example Embodiment)

In recent years, an image recognition technique using machine learningsuch as deep learning is applied to various systems. For example,application to a surveillance system for performing surveillance by animage of a surveillance camera has been advanced. By using machinelearning for the surveillance system, a state such as a pose andbehavior of a person is becoming recognizable from an image to someextent.

However, in such a related technique, a state of a person desired by auser may not be necessarily recognizable on demand. For example, thereis a case where a state of a person desired to be searched for andrecognized by a user can be determined in advance, or there is a casewhere a determination cannot be specifically made as in an unknownstate. Thus, in some cases, a state of a person desired to be searchedfor by a user cannot be specified in detail. Further, a search or thelike cannot be performed when a part of a body of a person is hidden. Inthe related technique, a state of a person can be searched for only froma specific search condition, and thus it is difficult to flexibly searchfor and classify a desired state of a person.

Thus, the inventors have considered a method using a skeleton estimationtechnique such as Non-Patent Document 1 and the like in order torecognize a state of a person desired by a user from an image on demand.Similarly to Open Pose disclosed in Non-Patent Document 1, and the like,in the related skeleton estimation technique, a skeleton of a person isestimated by learning image data in which correct answers in variouspatterns are set. In the following example embodiments, a state of aperson can be flexibly recognized by using such a skeleton estimationtechnique.

Note that, a skeleton structure estimated by the skeleton estimationtechnique such as Open Pose is formed of a “keypoint” being acharacteristic point such as a joint and a “bone (bone link)” indicatinga link between keypoints. Thus, in the following example embodiments,the words “keypoint” and “bone” will be used to describe a skeletonstructure, and “keypoint” is associated with a “joint” of a person and“bone” is associated with a “bone” of a person unless otherwisespecified. Then, a position of the “keypoint” is one example of jointinformation.

(Overview of Example Embodiment)

FIG. 1 illustrates an outline of an image processing apparatus 10according to an example embodiment. As illustrated in FIG. 1 , the imageprocessing apparatus 10 includes a skeleton detection unit 11, a featurevalue computation unit 12, and a recognition unit 13. The skeletondetection unit 11 detects two-dimensional skeleton structures of aplurality of persons, based on a two-dimensional image acquired from acamera and the like. The feature value computation unit 12 computesfeature values of the plurality of two-dimensional skeleton structuresdetected by the skeleton detection unit 11. The recognition unit 13performs recognition processing of a state of the plurality of persons,based on a degree of similarity between the plurality of feature valuescomputed by the feature value computation unit 12. The recognitionprocessing is classification processing, search processing (selectionprocessing), and the like of a state of a person. Thus, the imageprocessing apparatus 10 also functions as an image selection apparatus.

In this way, in the example embodiment, a two-dimensional skeletonstructure of a person is detected from a two-dimensional image, and therecognition processing such as classification and study of a state of aperson is performed based on a feature value computed from thetwo-dimensional skeleton structure, and thus a desired state of a personcan be flexibly recognized.

(Example Embodiment 1) An example embodiment 1 will be described belowwith reference to the drawings. FIG. 2 illustrates a configuration of animage processing apparatus 100 according to the present exampleembodiment. The image processing apparatus 100 constitutes an imageprocessing system 1, together with a camera 200 and a database (DB) 110.The image processing system 1 including the image processing apparatus100 is a system for classifying and searching for a state such as a poseand behavior of a person, based on a skeleton structure of the personestimated from an image. Note that, the image processing apparatus 100also functions as an image selection apparatus.

The camera 200 is an image capturing unit, such as a surveillancecamera, that generates a two-dimensional image. The camera 200 isinstalled at a predetermined place, and captures an image of a personand the like in the imaging area from the installed place. The camera200 may be directly connected to the image processing apparatus 100 insuch a way as to be able to output a captured image (video) to the imageprocessing apparatus 100, or may be connected to the image processingapparatus 100 via a network and the like. Note that, the camera 200 maybe provided inside the image processing apparatus 100.

The database 110 is a database that stores information (data) needed forprocessing of the image processing apparatus 100, a processing result,and the like. The database 110 stores an image acquired by an imageacquisition unit 101, a detection result of a skeleton structuredetection unit 102, data for machine learning, a feature value computedby a feature value computation unit 103, a classification result of aclassification unit 104, a search result of a search unit 105, and thelike. The database 110 is directly connected to the image processingapparatus 100 in such a way as to be able to input and output data asnecessary, or is connected to the image processing apparatus 100 via anetwork and the like. Note that, the database 110 may be provided insidethe image processing apparatus 100 as a non-volatile memory such as aflash memory, a hard disk apparatus, and the like.

As illustrated in FIG. 2 , the image processing apparatus 100 includesthe image acquisition unit 101, the skeleton structure detection unit102, the feature value computation unit 103, the classification unit104, the search unit 105, an input unit 106, and a display unit 107.Note that, a configuration of each unit (block) is one example, andanother unit may be used for a configuration as long as a method(operation) described below can be achieved. Further, the imageprocessing apparatus 100 is achieved by a computer apparatus, such as apersonal computer and a server, that executes a program, for example,but may be achieved by one apparatus or may be achieved by a pluralityof apparatuses on a network. For example, the input unit 106, thedisplay unit 107, and the like may be an external apparatus. Further,both of the classification unit 104 and the search unit 105 may beprovided, or only one of them may be provided. Both or one of theclassification unit 104 and the search unit 105 is a recognition unitthat performs the recognition processing of a state of a person.

The image acquisition unit 101 acquires a two-dimensional imageincluding a person captured by the camera 200. The image acquisitionunit 101 acquires an image (video including a plurality of images)including a person captured by the camera 200 in a predeterminedsurveillance period, for example. Note that, instead of acquisition fromthe camera 200, an image including a person being prepared in advancemay be acquired from the database 110 and the like.

The skeleton structure detection unit 102 detects a two-dimensionalskeleton structure of the person in the acquired two-dimensional image,based on the image. The skeleton structure detection unit 102 detects askeleton structure for all persons recognized in the acquired image. Theskeleton structure detection unit 102 detects a skeleton structure of arecognized person, based on a feature such as a joint of the person, byusing a skeleton estimation technique using machine learning. Theskeleton structure detection unit 102 uses a skeleton estimationtechnique such as Open Pose in Non-Patent Document 1, for example.

The feature value computation unit 103 computes a feature value of thedetected two-dimensional skeleton structure, and stores, in the database110, the computed feature value in association with the image to beprocessed. The feature value of the skeleton structure indicates afeature of a skeleton of the person, and is an element for classifyingand searching for a state of the person, based on the skeleton of theperson. This feature value normally includes a plurality of parameters(for example, a classification element described below). Then, thefeature value may be a feature value of the entire skeleton structure,may be a feature value of a part of the skeleton structure, or mayinclude a plurality of feature values as in each portion of the skeletonstructure. A method for computing a feature value may be any method suchas machine learning and normalization, and a minimum value and a maximumvalue may be acquired as normalization. As one example, the featurevalue is a feature value acquired by performing machine learning on theskeleton structure, a size of the skeleton structure from a head to afoot on an image, and the like. The size of the skeleton structure is aheight in an up-down direction, an area, and the like of a skeletonregion including the skeleton structure on an image. The up-downdirection (a height direction or a vertical direction) is a direction(Y-axis direction) of up and down in an image, and is, for example, adirection perpendicular to the ground (reference surface). Further, aleft-right direction (a horizontal direction) is a direction (X-axisdirection) of left and right in an image, and is, for example, adirection parallel to the ground.

Note that, in order to perform classification and a search desired by auser, a feature value having robustness with respect to classificationand search processing is preferably used. For example, when a userdesires classification and a search that do not depend on an orientationand a body shape of a person, a feature value that is robust withrespect to the orientation and the body shape of the person may be used.A feature value that does not depend on an orientation and a body shapeof a person can be acquired by learning skeletons of persons facing invarious directions with the same pose and skeletons of persons havingvarious body shapes with the same pose, and extracting a feature only inthe up-down direction of a skeleton.

The classification unit 104 classifies a plurality of skeletonstructures stored in the database 110, based on a degree of similaritybetween feature values of the skeleton structures (performs clustering).It can also be said that, as the recognition processing of a state of aperson, the classification unit 104 classifies states of a plurality ofpersons, based on feature values of the skeleton structures. The degreeof similarity is a distance between the feature values of the skeletonstructures. The classification unit 104 may perform classification by adegree of similarity between feature values of the entire skeletonstructures, may perform classification by a degree of similarity betweenfeature values of a part of the skeleton structures, and may performclassification by a degree of similarity between feature values of afirst portion (for example, both hands) and a second portion (forexample, both feet) of the skeleton structures. Note that, a pose of aperson may be classified based on a feature value of a skeletonstructure of the person in each image, and behavior of a person may beclassified based on a change in a feature value of a skeleton structureof the person in a plurality of images successive in time series. Inother words, the classification unit 104 may classify a state of aperson including a pose and behavior of the person, based on a featurevalue of a skeleton structure. For example, the classification unit 104sets, as subjects to be classified, a plurality of skeleton structuresin a plurality of images captured in a predetermined surveillanceperiod. The classification unit 104 acquires a degree of similaritybetween feature values of the subjects to be classified, and performsclassification in such a way that skeleton structures having a highdegree of similarity are in the same cluster (group with a similarpose). Note that, similarly to a search, a user may be able to specify aclassification condition. The classification unit 104 stores aclassification result of the skeleton structure in the database 110, andalso displays the classification result on the display unit 107.

The search unit 105 searches for a skeleton structure having a highdegree of similarity to a feature value of a search query (query state)from among the plurality of skeleton structures stored in the database110. It can also be said that, as the recognition processing of a stateof a person, the search unit 105 searches for a state of a person thatcorresponds to a search condition (query state) from among states of aplurality of persons, based on feature values of the skeletonstructures. Similarly to classification, the degree of similarity is adistance between the feature values of the skeleton structures. Thesearch unit 105 may perform a search by a degree of similarity betweenfeature values of the entire skeleton structures, may perform a searchby a degree of similarity between feature values of a part of theskeleton structures, and may perform a search by a degree of similaritybetween feature values of a first portion (for example, both hands) anda second portion (for example, both feet) of the skeleton structures.Note that, a pose of a person may be searched based on a feature valueof a skeleton structure of the person in each image, and behavior of aperson may be searched based on a change in a feature value of askeleton structure of the person in a plurality of images successive intime series. In other words, the search unit 105 can search for a stateof a person including a pose and behavior of the person, based on afeature value of a skeleton structure. For example, similarly tosubjects to be classified, the search unit 105 sets, as subjects to besearched, feature values of a plurality of skeleton structures in aplurality of images captured in a predetermined surveillance period.Further, a skeleton structure (pose) specified by a user from amongclassification results displayed on the classification unit 104 is setas a search query (search key). Note that, without limitation to aclassification result, a search query may be selected from among aplurality of skeleton structures that are not classified, or a user mayinput a skeleton structure to be a search query. The search unit 105searches for a feature value having a high degree of similarity to afeature value of a skeleton structure being a search query from amongfeature values being subjects to be searched. The search unit 105 storesa search result of the feature value in the database 110, and alsodisplays the search result on the display unit 107.

The input unit 106 is an input interface that acquires information inputby a user who operates the image processing apparatus 100. For example,the user is a surveillant who watches a person in a suspicious statefrom an image of a surveillance camera. The input unit 106 is, forexample, a graphical user interface (GUI), and receives an input ofinformation according to an operation of the user from an inputapparatus such as a keyboard, a mouse, and a touch panel. For example,the input unit 106 receives, as a search query, a skeleton structure ofa person specified from among the skeleton structures (poses) classifiedby the classification unit 104.

The display unit 107 is a display unit that displays a result of anoperation (processing) of the image processing apparatus 100, and thelike, and is, for example, a display apparatus such as a liquid crystaldisplay and an organic electro luminescence (EL) display. The displayunit 107 displays, on the GUI, a classification result of theclassification unit 104 and a search result of the search unit 105according to a degree of similarity and the like.

FIG. 39 is a diagram illustrating a hardware configuration example ofthe image processing apparatus 100. The image processing apparatus 100includes a bus 1010, a processor 1020, a memory 1030, a storage device1040, an input/output interface 1050, and a network interface 1060.

The bus 1010 is a data transmission path for allowing the processor1020, the memory 1030, the storage device 1040, the input/outputinterface 1050, and the network interface 1060 to transmit and receivedata with one another. However, a method of connecting the processor1020 and the like to each other is not limited to bus connection.

The processor 1020 is a processor achieved by a central processing unit(CPU), a graphics processing unit (GPU), and the like.

The memory 1030 is a main storage achieved by a random access memory(RAM) and the like.

The storage device 1040 is an auxiliary storage achieved by a hard diskdrive (HDD), a solid state drive (SSD), a memory card, a read onlymemory (ROM), or the like. The storage device 1040 stores a programmodule that achieves each function (for example, the image acquisitionunit 101, the skeleton structure detection unit 102, the feature valuecomputation unit 103, the classification unit 104, the search unit 105,and the input unit 106) of the image processing apparatus 100. Theprocessor 1020 reads each program module onto the memory 1030 andexecutes the program module, and each function associated with theprogram module is achieved. Further, the storage device 1040 may alsofunction as the database 110.

The input/output interface 1050 is an interface for connecting the imageprocessing apparatus 100 and various types of input/output equipment.When the database 110 is located outside the image processing apparatus100, the image processing apparatus 100 may be connected to the database110 via the input/output interface 1050.

The network interface 1060 is an interface for connecting the imageprocessing apparatus 100 to a network. The network is, for example, alocal area network (LAN) and a wide area network (WAN). A method ofconnection to the network by the network interface 1060 may be wirelessconnection or wired connection. The image processing apparatus 100 maycommunicate with the camera 200 via the network interface 1060. When thedatabase 110 is located outside the image processing apparatus 100, theimage processing apparatus 100 may be connected to the database 110 viathe network interface 1060.

FIGS. 3 to 5 illustrate operations of the image processing apparatus 100according to the present example embodiment. FIG. 3 illustrates a flowfrom image acquisition to search processing in the image processingapparatus 100, FIG. 4 illustrates a flow of classification processing(S104) in FIG. 3 , and FIG. 5 illustrates a flow of the searchprocessing (S105) in FIG. 3 .

As illustrated in FIG. 3 , the image processing apparatus 100 acquiresan image from the camera 200 (S101). The image acquisition unit 101acquires an image in which a person is captured for performingclassification and a search based on a skeleton structure, and storesthe acquired image in the database 110. For example, the imageacquisition unit 101 acquires a plurality of images captured in apredetermined surveillance period, and performs the following processingon all persons included in the plurality of images.

Subsequently, the image processing apparatus 100 detects a skeletonstructure of a person, based on the acquired image of the person (S102).FIG. 6 illustrates a detection example of skeleton structures. Asillustrated in FIG. 6 , a plurality of persons are included in an imageacquired from a surveillance camera or the like, and a skeletonstructure is detected for each of the persons included in the image.

FIG. 7 illustrates a skeleton structure of a human model 300 detected atthis time, and FIGS. 8 to 10 each illustrate a detection example of theskeleton structure. The skeleton structure detection unit 102 detectsthe skeleton structure of the human model (two-dimensional skeletonmodel) 300 as in FIG. 7 from a two-dimensional image by using a skeletonestimation technique such as Open Pose. The human model 300 is atwo-dimensional model formed of a keypoint such as a joint of a personand a bone connecting keypoints.

For example, the skeleton structure detection unit 102 extracts afeature point that may be a keypoint from an image, refers toinformation acquired by performing machine learning on the image of thekeypoint, and detects each keypoint of a person. In the exampleillustrated in FIG. 7 , as a keypoint of a person, a head A1, a neck A2,a right shoulder A31, a left shoulder A32, a right elbow A41, a leftelbow A42, a right hand A51, a left hand A52, a right waist A61, a leftwaist A62, a right knee A71, a left knee A72, a right foot A81, and aleft foot A82 are detected. Furthermore, as a bone of the personconnecting the keypoints, detected are a bone B1 connecting the head A1and the neck A2, a bone B21 connecting the neck A2 and the rightshoulder A31, a bone B22 connecting the neck A2 and the left shoulderA32, a bone B31 connecting the right shoulder A31 and the right elbowA41, a bone B32 connecting the left shoulder A32 and the left elbow A42,a bone B41 connecting the right elbow A41 and the right hand A51, a boneB42 connecting the left elbow A42 and the left hand A52, a bone B51connecting the neck A2 and the right waist A61, a bone B52 connectingthe neck A2 and the left waist A62, a bone B61 connecting the rightwaist A61 and the right knee A71, a bone B62 connecting the left waistA62 and the left knee A72, a bone B71 connecting the right knee A71 andthe right foot A81, and a bone B72 connecting the left knee A72 and theleft foot A82. The skeleton structure detection unit 102 stores thedetected skeleton structure of the person in the database 110.

FIG. 8 is an example of detecting a person in an upright state. In FIG.8 , an image of the upright person is captured from the front, the boneB1, the bone B51 and the bone B52, the bone B61 and the bone B62, andthe bone B71 and the bone B72 that are viewed from the front are eachdetected without overlapping, and the bone B61 and the bone B71 of aright leg are bent slightly more than the bone B62 and the bone B72 of aleft leg.

FIG. 9 is an example of detecting a person in a squatting state. In FIG.9 , an image of the squatting person is captured from a right side, thebone B1, the bone B51 and the bone B52, the bone B61 and the bone B62,and the bone B71 and the bone B72 that are viewed from the right sideare each detected, and the bone B61 and the bone B71 of a right leg andthe bone B62 and the bone B72 of a left leg are greatly bent and alsooverlap.

FIG. 10 is an example of detecting a person in a sleeping state. In FIG.10, an image of the sleeping person is captured diagonally from thefront left, the bone B1, the bone B51 and the bone B52, the bone B61 andthe bone B62, and the bone B71 and the bone B72 that are vieweddiagonally from the front left are each detected, and the bone B61 andthe bone B71 of a right leg, and the bone B62 and the bone B72 of a leftleg are bent and also overlap.

Subsequently, as illustrated in FIG. 3 , the image processing apparatus100 computes a feature value of the detected skeleton structure (S103).For example, when a height and an area of a skeleton region are set afeature value, the feature value computation unit 103 extracts a regionincluding the skeleton structure and acquires a height (pixel count) andan area (pixel area) of the region. The height and the area of theskeleton region are acquired from coordinates of an end portion of theextracted skeleton region and coordinates of a keypoint of the endportion. The feature value computation unit 103 stores the acquiredfeature value of the skeleton structure in the database 110.

Note that, the feature value of the skeleton structure is also used aspose information indicating a pose of the person. Further, the featurevalue of the skeleton structure is stored in the database 110 togetherwith a degree of reliability of the feature value. The degree ofreliability of the feature value indicates a likelihood (i.e., aprobability that an estimation result of a pose is correct) of being thefeature value. When a degree of reliability of a certain feature valueis high, a probability of being the feature value is high. Note that, adegree of reliability of a feature value is computed by using, forexample, the number of keypoints used when the feature value of askeleton structure is set, and a degree of reliability of the keypoint.For example, a greater number of joints increases a degree ofreliability of a feature value. Furthermore, a higher degree ofreliability of a keypoint increases a degree of reliability of a featurevalue.

In the example in FIG. 8 , a skeleton region including all of the bonesis extracted from the skeleton structure of the upright person. In thiscase, an upper end of the skeleton region is the keypoint A1 of thehead, a lower end of the skeleton region is the keypoint A82 of the leftfoot, a left end of the skeleton region is the keypoint A41 of the rightelbow, and a right end of the skeleton region is the keypoint A52 of theleft hand. Thus, a height of the skeleton region is acquired from adifference in Y coordinate between the keypoint A1 and the keypoint A82.Further, a width of the skeleton region is acquired from a difference inX coordinate between the keypoint A41 and the keypoint A52, and an areais acquired from the height and the width of the skeleton region.

In the example in FIG. 9 , a skeleton region including all of the bonesis extracted from the skeleton structure of the squatting person. Inthis case, an upper end of the skeleton region is the keypoint A1 of thehead, a lower end of the skeleton region is the keypoint A81 of theright foot, a left end of the skeleton region is the keypoint A61 of theright waist, and a right end of the skeleton region is the keypoint A51of the right hand. Thus, a height of the skeleton region is acquiredfrom a difference in Y coordinate between the keypoint A1 and thekeypoint A81. Further, a width of the skeleton region is acquired from adifference in X coordinate between the keypoint A61 and the keypointA51, and an area is acquired from the height and the width of theskeleton region.

In the example in FIG. 10 , a skeleton region including all of the bonesis extracted from the skeleton structure of the sleeping person lyingalong the left-right direction of the image. In this case, an upper endof the skeleton region is the keypoint A32 of the left shoulder, a lowerend of the skeleton region is the keypoint A52 of the left hand, a leftend of the skeleton region is the keypoint A51 of the right hand, and aright end of the skeleton region is the keypoint A82 of the left foot.Thus, a height of the skeleton region is acquired from a difference in Ycoordinate between the keypoint A32 and the keypoint A52. Further, awidth of the skeleton region is acquired from a difference in Xcoordinate between the keypoint A51 and the keypoint A82, and an area isacquired from the height and the width of the skeleton region.

Subsequently, as illustrated in FIG. 3 , the image processing apparatus100 performs classification processing (S104). In the classificationprocessing, as illustrated in FIG. 4 , the classification unit 104computes a degree of similarity of the computed feature value of theskeleton structure (S111), and classifies the skeleton structure basedon the computed feature value (S112). The classification unit 104acquires a degree of similarity among all of the skeleton structuresthat are subjects to be classified and are stored in the database 110,and classifies skeleton structures (poses) having a highest degree ofsimilarity in the same cluster (performs clustering). Furthermore,classification is performed by acquiring a degree of similarity betweenclassified clusters, and classification is repeated until the number ofclusters becomes a predetermined number. FIG. 11 illustrates an image ofa classification result of feature values of skeleton structures. FIG.11 is an image of a cluster analysis by two-dimensional classificationelements, and two classification elements are, for example, a height ofa skeleton region and an area of the skeleton region, or the like. InFIG. 11 , as a result of classification, feature values of a pluralityof skeleton structures are classified into three clusters C1 to C3. Theclusters C1 to C3 are associated with poses such as a standing pose, asitting pose, and a sleeping pose, respectively, for example, andskeleton structures (persons) are classified for each similar pose.

In the present example embodiment, various classification methods can beused by performing classification, based on a feature value of askeleton structure of a person. Note that, a classification method maybe preset, or any classification method may be able to be set by a user.Further, classification may be performed by the same method as a searchmethod described below. In other words, classification may be performedby a classification condition similar to a search condition. Forexample, the classification unit 104 performs classification by thefollowing classification methods. Any classification method may be used,or any selected classification methods may be combined.

(Classification Method 1) Classification by a Plurality of HierarchicalLevels

Classification is performed by combining, in a hierarchical manner,classification by a skeleton structure of a whole body, classificationby a skeleton structure of an upper body and a lower body,classification by a skeleton structure of an arm and a leg, and thelike. In other words, classification may be performed based on a featurevalue of a first portion and a second portion of a skeleton structure,and, furthermore, classification may be performed by assigning weightsto the feature value of the first portion and the second portion.

(Classification Method 2) Classification by a Plurality of Images AlongTime Series

Classification is performed based on a feature value of a skeletonstructure in a plurality of images successive in time series. Forexample, classification may be performed based on a cumulative value byaccumulating a feature value in a time series direction. Furthermore,classification may be performed based on a change (change value) in afeature value of a skeleton structure in a plurality of successiveimages.

(Classification Method 3) Classification by Ignoring the Left and theRight of a Skeleton Structure

Classification is performed on an assumption that skeleton structures inwhich a right side and a left side are reversed are the same skeletonstructure.

Furthermore, the classification unit 104 displays a classificationresult of the skeleton structure (S113). The classification unit 104acquires a necessary image of a skeleton structure and a person from thedatabase 110, and displays, on the display unit 107, the skeletonstructure and the person for each similar pose (cluster) as aclassification result. FIG. 12 illustrates a display example when posesare classified into three. For example, as illustrated in FIG. 12 , poseregions WA1 to WA3 for each pose are displayed on a display window W1,and a skeleton structure and a person (image) of each associated poseare displayed in the pose regions WA1 to WA3. The pose region WA1 is,for example, a display region of a standing pose, and displays askeleton structure and a person that are classified into the cluster C1and are similar to the standing pose. The pose region WA2 is, forexample, a display region of a sitting pose, and displays a skeletonstructure and a person that are classified into the cluster C2 and aresimilar to the sitting pose. The pose region WA3 is, for example, adisplay region of a sleeping pose, and displays a skeleton structure anda person that are classified into the cluster C2 and are similar to thesleeping pose.

Subsequently, as illustrated in FIG. 3 , the image processing apparatus100 performs the search processing (S105). In the search processing, asillustrated in FIG. 5 , the search unit 105 receives an input of asearch condition (S121), and searches for a skeleton structure, based onthe search condition (S122). The search unit 105 receives, from theinput unit 106, an input of a search query being the search condition inresponse to an operation of a user. When the search query is input froma classification result, for example, in the display example in FIG. 12, a user specifies (selects), from among the pose regions WA1 to WA3displayed on the display window W1, a skeleton structure of a posedesired to be searched for. Then, with the skeleton structure specifiedby the user as the search query, the search unit 105 searches for askeleton structure having a high degree of similarity of a feature valuefrom among all of the skeleton structures that are subjects to besearched and are stored in the database 110. The search unit 105computes a degree of similarity between a feature value of the skeletonstructure being the search query and a feature value of the skeletonstructure being the subject to be searched, and extracts a skeletonstructure having the computed degree of similarity higher than apredetermined threshold value. The feature value of the skeletonstructure being the search query may use a feature value being computedin advance, or may use a feature value being acquired during a search.Note that, the search query may be input by moving each portion of askeleton structure in response to an operation of the user, or a posedemonstrated by the user in front of a camera may be set as the searchquery.

In the present example embodiment, similarly to the classificationmethods, various search methods can be used by performing a search,based on a feature value of a skeleton structure of a person. Note that,a search method may be preset, or any search method may be able to beset by a user. For example, the search unit 105 performs a search by thefollowing search methods. Any search method may be used, or any selectedsearch methods may be combined. A search may be performed by combining aplurality of search methods (search conditions) by a logical expression(for example, AND (conjunction), OR (disjunction), NOT (negation)). Forexample, a search may be performed by setting “(pose with a right handup) AND (pose with a left foot up)” as a search condition.

(Search Method 1) A search only by a feature value in the heightdirection By performing a search by using only a feature value in theheight direction of a search person, an influence of a change in thehorizontal direction of a person can be suppressed, and robustnessimproves with respect to a change in orientation of the person and bodyshape of the person. For example, as in skeleton structures 501 to 503in FIG. 13 , even when there is difference in an orientation or a bodyshape of a person, a feature value in the height direction does notgreatly change. Thus, in the skeleton structures 501 to 503, it can bedecided, at a time of a search (at a time of classification), that posesare the same.

(Search Method 2) When a part of a body of a person is hidden in apartial search image, a search is performed by using only informationabout a recognizable portion. For example, as in skeleton structures 511and 512 in FIG. 14 , even when a keypoint of a left foot cannot bedetected due to the left foot being hidden, a search can be performed byusing a feature value of another detected keypoint. Thus, in theskeleton structures 511 and 512, it can be decided, at a time of asearch (at a time of classification), that poses are the same. In otherwords, classification and a search can be performed by using a featurevalue of some of keypoints instead of all keypoints. In an example ofskeleton structures 521 and 522 in FIG. 15 , although orientations ofboth feet are different, it can be decided that poses are the same bysetting a feature value of keypoints (A1, A2, A31, A32, A41, A42, A51,and A52) of an upper body as a search query. Further, a search may beperformed by assigning a weight to a portion (feature point) desired tobe searched, or a threshold value of a similarity degree determinationmay be changed. When a part of a body is hidden, a search may beperformed by ignoring the hidden portion, or a search may be performedby taking the hidden portion into consideration. By performing a searchalso including a hidden portion, a pose in which the same portion ishidden can be searched.

(Search Method 3) Search by Ignoring the Left and the Right of aSkeleton Structure

A search is performed on an assumption that skeleton structures in whicha right side and a left side are reversed are the same skeletonstructure. For example, as in skeleton structures 531 and 532 in FIG. 16, a pose with a right hand up and a pose with a left hand up can besearched (classified) as the same pose. In the example in FIG. 16 , inthe skeleton structure 531 and the skeleton structure 532, althoughpositions of the keypoint A51 of the right hand, the keypoint A41 of theright elbow, the keypoint A52 of the left hand, and the keypoint A42 ofthe left elbow are different, positions of the other keypoints are thesame. When the keypoints of one of the skeleton structures, of thekeypoint A51 of the right hand and the keypoint A41 of the right elbowof the skeleton structure 531 and the keypoint A52 of the left hand andthe keypoint A42 of the left elbow of the skeleton structure 532, arereversed, the keypoints have the same positions of the keypoints of theother skeleton structure. When the keypoints of one of the skeletonstructures, of the keypoint A52 of the left hand and the keypoint A42 ofthe left elbow of the skeleton structure 531 and the keypoint A51 of theright hand and the keypoint A41 of the right elbow of the skeletonstructure 532, are reversed, the keypoints have the same positions ofthe keypoints of the other skeleton structure. Thus, it is decided thatposes are the same.

(Search Method 4) A Search by a Feature Value in the Vertical Directionand the Horizontal Direction

After a search is performed only with a feature value of a person in thevertical direction (Y-axis direction), the acquired result is furthersearched by using a feature value of the person in the horizontaldirection (X-axis direction).

(Search Method 5) a Search by a Plurality of Images Along Time Series

A search is performed based on a feature value of a skeleton structurein a plurality of images successive in time series. For example, asearch may be performed based on a cumulative value by accumulating afeature value in a time series direction. Furthermore, a search may beperformed based on a change (change value) in a feature value of askeleton structure in a plurality of successive images.

Furthermore, the search unit 105 displays a search result of theskeleton structure (S123). The search unit 105 acquires a necessaryimage of a skeleton structure and a person from the database 110, anddisplays, on the display unit 107, the skeleton structure and the personacquired as a search result. For example, when a plurality of searchqueries (search conditions) are specified, a search result is displayedfor each of the search queries. FIG. 17 illustrates a display examplewhen a search is performed by three search queries (poses). For example,as illustrated in FIG. 17 , in a display window W2, skeleton structuresand persons of search queries Q10, Q20, and Q30 specified are displayedat a left end portion, and skeleton structures and persons of searchresults Q11, Q21, and Q31 of the search queries are displayed side byside on the right side of the search queries Q10, Q20, and Q30.

An order in which search results are displayed side by side from asearch query may be an order in which a corresponding skeleton structureis found, or may be decreasing order of a degree of similarity. When asearch is performed by assigning a weight to a portion (feature point)in a partial search, display may be performed in an order of a degree ofsimilarity computed by assigning a weight. Display may be performed inan order of a degree of similarity computed only from a portion (featurepoint) selected by a user. Further, display may be performed by cutting,for a certain period of time, images (frames) in time series before andafter an image (frame) that is a search result.

(Search Method 6) In the present search method, a plurality of searchedimages are set by using a degree of reliability of pose informationabout the images.

FIG. 40 is a diagram illustrating one example of a functionalconfiguration of the search unit 105 according to the present searchmethod. In the example illustrated in FIG. 40 , the search unit 105includes a query acquisition unit 610, a search information acquisitionunit 620, a selection unit 630, and a display control unit 640.

The query acquisition unit 610 acquires pose information (hereinafter,described as query pose information) that is information to be a queryand indicates a pose of a person.

The search information acquisition unit 620 acquires a plurality ofpieces of search pose information. The search pose information is poseinformation about an image (hereinafter, described as a target image) tobe a search target, and is stored in the database 110 for each of targetimages.

The selection unit 630 selects, from among the plurality of pieces ofsearch pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference. The selection processing is substantiallyprocessing of selecting a target image similar to an image (hereinafter,described as a query image) associated with the query pose information.

The display control unit 640 displays, on the display unit 107, targetimages associated with the respective two or more pieces of the searchpose information selected by the selection unit 630, and also setsdisplay positions of the target images by using a degree of reliabilityof the search pose information. The degree of reliability is stored inthe database 110 as described above.

FIG. 41 is a flowchart illustrating one example of processing performedby the search unit 105 according to the present search method. First,the query acquisition unit 610 acquires query pose information (stepS300). As one example, the query acquisition unit 610 acquires the querypose information from the database 110. In this case, for example, auser performs an input for determining a query image on the queryacquisition unit 610. Then, the query acquisition unit 610 reads thequery pose information associated with the query image from the database110.

Further, the query acquisition unit 610 may acquire an image to be aquery from an external apparatus. In this case, the skeleton structuredetection unit 102 and the feature value computation unit 103 processthe image and compute a feature value of a skeleton structure. Then, thequery acquisition unit 610 acquires the feature value of the skeletonstructure as the query pose information.

Next, the search information acquisition unit 620 acquires, as searchinformation, a plurality of pieces of pose information stored in thedatabase 110 (step S310). Next, the selection unit 630 selects, fromamong a plurality of pieces of search pose information, two or morepieces of search pose information whose degree of similarity to thequery pose information satisfies a reference (step S320). The processingperformed herein is performed by using at least one of the searchmethods 1 to 5 described above, for example, but may be performed byusing another method.

Next, the display control unit 640 reads a target image associated withthe search pose information selected in step S320 from the database 110.Since the two or more pieces of the search pose information are selectedin step S320, the display control unit 640 also reads two or more targetimages from the database 110. At this time, the display control unit 640also reads a degree of reliability of each of the target images from thedatabase 110 (step S330).

Next, the display control unit 640 sets a display position of each ofthe target images by using the degree of reliability of the target image(step S340). As described above, the degree of reliability indicates aprobability that an estimation result of a pose is correct. Thus, whenthe degree of reliability is high, a probability that the target imageis similar to a query image is high. Therefore, when a display positionof the target image is set by using the degree of reliability of thetarget image, an influence of the degree of reliability of each of thetarget images on appropriateness of the selection result in step S320can be recognized by the user.

Then, the display control unit 640 displays the target image read instep S330 on the display unit 107. At this time, the display controlunit 640 displays each of the target images in the display positiondetermined in step S340 (step S350).

Further, the selection unit 630 stores the selection result in step S320in the database 110 (step S360). Herein, the selection unit 630 maystore the target image read in step S330 as one cluster in the database110, or may associate, with the target image already stored in thedatabase 110, a flag indicating that the target image is similar to thequery image.

FIG. 42 is a diagram illustrating one example of a method fordetermining a display position of a target image being performed in stepS340. In the example illustrated in FIG. 42 , the display control unit640 arranges target images in decreasing order of a degree ofreliability, and displays the target images on the display unit 107.However, the arrangement order of the target images is not limited tothis example.

Further, the display control unit 640 displays, on the display unit 107,the target image together with a degree of reliability of search poseinformation of the target image. In such a manner, a user can directlyconfirm a degree of reliability of a selected target image, and can thusrecognize a degree of reliability needed to acquire an appropriateselection result.

FIG. 43 is a flowchart illustrating a modification example 1 of FIG. 41. The example illustrated in FIG. 43 is similar to FIG. 41 except for apoint that the display control unit 640 also uses a degree of similaritybetween a target image and a query image, i.e., a degree of similaritybetween search pose information and query pose information in additionto a degree of reliability of the target image when a display positionof the target image is determined (step S342).

As one example, the display control unit 640 determines an arrangementorder of target images by using the degree of reliability and the degreeof similarity described above. For example, the display control unit 640computes an evaluation value by using a function with the degree ofreliability and the degree of similarity described above as a parameter,and displays target images in decreasing order (or ascending order) ofthe evaluation value. One example of the function is a function inwhich, for example, a predetermined weight coefficient is added to eachof the degree of reliability and the degree of similarity, and a sum ofthem is then computed.

FIG. 44 is a flowchart illustrating a modification example 2 of FIG. 41. The example illustrated in FIG. 44 is similar to FIG. 41 except for apoint that the display control unit 640 also uses, with a condition thata predetermined condition is satisfied (step S332: Yes), a degree ofsimilarity between a target image and a query image in addition to adegree of reliability of the target image or instead of the degree ofreliability when a display position of the target image is determined(step S344).

Processing when a degree of similarity between a target image and aquery image is used in addition to a degree of reliability of the targetimage is similar to step S342 in FIG. 43 . Further, processing when adegree of similarity between a target image and a query image is usedinstead of a degree of reliability of the target image is similar tostep S340 in FIG. 41 except for a point that the degree of similarity isused.

As described above, in the present example embodiment, a skeletonstructure of a person can be detected from a two-dimensional image, andclassification and a search can be performed based on a feature value ofthe detected skeleton structure. In this way, classification can beperformed for each similar pose having a high degree of similarity, anda similar pose having a high degree of similarity to a search query(search key) can be searched. By classifying similar poses from an imageand displaying the similar poses, a user can recognize a pose of aperson in the image without specifying a pose and the like. Since theuser can specify a pose being a search query from a classificationresult, a desired pose can be searched for even when a pose desired tobe searched for by a user is not recognized in detail in advance. Forexample, since classification and a search can be performed with a wholeor a part of a skeleton structure of a person and the like as acondition, flexible classification and a flexible search can beperformed.

Further, according to the search method 6, when at least two images(target images) are selected, a display position of each target image isset by using a degree of reliability of the target image. Thus, aninfluence of the degree of reliability of each of the target images onappropriateness of a selection result can be recognized by a user.

(Example Embodiment 2) An example embodiment 2 will be described belowwith reference to the drawings. In the present example embodiment, aspecific example of the feature value computation in the exampleembodiment 1 will be described. In the present example embodiment, afeature value is acquired by normalization by using a height of aperson. The other points are similar to those in the example embodiment1.

FIG. 18 illustrates a configuration of an image processing apparatus 100according to the present example embodiment. As illustrated in FIG. 18 ,the image processing apparatus 100 further includes a height computationunit 108 in addition to the configuration in the example embodiment 1.Note that, a feature value computation unit 103 and the heightcomputation unit 108 may serve as one processing unit.

The height computation unit (height estimation unit) 108 computes(estimates) an upright height (referred to as a height pixel count) of aperson in a two-dimensional image, based on a two-dimensional skeletonstructure detected by a skeleton structure detection unit 102. It can besaid that the height pixel count is a height of a person in atwo-dimensional image (a length of a whole body of a person on atwo-dimensional image space). The height computation unit 108 acquires aheight pixel count (pixel count) from a length (length on thetwo-dimensional image space) of each bone of a detected skeletonstructure.

In the following examples, specific examples 1 to 3 are used as a methodfor acquiring a height pixel count. Note that, any method of thespecific examples 1 to 3 may be used, or a plurality of any selectedmethods may be combined and used. In the specific example 1, a heightpixel count is acquired by adding up lengths of bones from a head to afoot among bones of a skeleton structure. When the skeleton structuredetection unit 102 (skeleton estimation technique) does not output a topof a head and a foot, a correction can be performed by multiplication bya constant as necessary. In the specific example 2, a height pixel countis computed by using a human model indicating a relationship between alength of each bone and a length of a whole body (a height on thetwo-dimensional image space). In the specific example 3, a height pixelcount is computed by fitting (applying) a three-dimensional human modelto a two-dimensional skeleton structure.

The feature value computation unit 103 according to the present exampleembodiment is a normalization unit that normalizes a skeleton structure(skeleton information) of a person, based on a computed height pixelcount of the person. The feature value computation unit 103 stores afeature value (normalization value) of the normalized skeleton structurein a database 110. The feature value computation unit 103 normalizes, bythe height pixel count, a height on an image of each keypoint (featurepoint) included in the skeleton structure. In the present exampleembodiment, for example, a height direction is an up-down direction(Y-axis direction) in a two-dimensional coordinate (X-Y coordinate)space of an image. In this case, a height of a keypoint can be acquiredfrom a value (pixel count) of a Y coordinate of the keypoint.Alternatively, a height direction may be a direction (verticalprojection direction) of a vertical projection axis in which a directionof a vertical axis perpendicular to the ground (reference surface) in athree-dimensional coordinate space in a real world is projected in thetwo-dimensional coordinate space. In this case, a height of a keypointcan be acquired from a value (pixel count) along a vertical projectionaxis, the vertical projection axis being acquired by projecting an axisperpendicular to the ground in the real world to the two-dimensionalcoordinate space, based on a camera parameter. Note that, the cameraparameter is a capturing parameter of an image, and, for example, thecamera parameter is a pose, a position, a capturing angle, a focaldistance, and the like of a camera 200. The camera 200 captures an imageof an object whose length and position are clear in advance, and acamera parameter can be acquired from the image. A strain may occur atboth ends of the captured image, and the vertical direction in the realworld and the up-down direction in the image may not match. In contrast,an extent that the vertical direction in the real world is tilted in animage is clear by using a parameter of a camera that captures the image.Thus, a feature value of a keypoint can be acquired in consideration ofa difference between the real world and the image by normalizing, by aheight, a value of the keypoint along a vertical projection axisprojected in the image, based on the camera parameter. Note that, aleft-right direction (a horizontal direction) is a direction (X-axisdirection) of left and right in a two-dimensional coordinate (X-Ycoordinate) space of an image, or is a direction in which a directionparallel to the ground in the three-dimensional coordinate space in thereal world is projected to the two-dimensional coordinate space.

FIGS. 19 to 23 illustrate operations of the image processing apparatus100 according to the present example embodiment. FIG. 19 illustrates aflow from image acquisition to search processing in the image processingapparatus 100, FIGS. 20 to 22 illustrate flows of specific examples 1 to3 of height pixel count computation processing (S201) in FIG. 19 , andFIG. 23 illustrates a flow of normalization processing (S202) in FIG. 19.

As illustrated in FIG. 19 , in the present example embodiment, theheight pixel count computation processing (S201) and the normalizationprocessing (S202) are performed as the feature value computationprocessing (S103) in the example embodiment 1. The other points aresimilar to those in the example embodiment 1.

The image processing apparatus 100 performs the height pixel countcomputation processing (S201), based on a detected skeleton structure,after the image acquisition (S101) and skeleton structure detection(S102). In this example, as illustrated in FIG. 24 , a height of askeleton structure of an upright person in an image is a height pixelcount (h), and a height of each keypoint of the skeleton structure inthe state of the person in the image is a keypoint height (yi).Hereinafter, the specific examples 1 to 3 of the height pixel countcomputation processing will be described.

<Specific Example 1> In the specific example 1, a height pixel count isacquired by using a length of a bone from a head to a foot. In thespecific example 1, as illustrated in FIG. 20 , the height computationunit 108 acquires a length of each bone (S211), and adds up the acquiredlength of each bone (S212).

The height computation unit 108 acquires a length of a bone from a headto a foot of a person on a two-dimensional image, and acquires a heightpixel count. In other words, each length (pixel count) of a bone B1(length L1), a bone B51 (length L21), a bone B61 (length L31), and abone B71 (length L41), or the bone B1 (length L1), a bone B52 (lengthL22), a bone B62 (length L32), and a bone B72 (length L42) among bonesin FIG. 24 is acquired from the image in which the skeleton structure isdetected. A length of each bone can be acquired from coordinates of eachkeypoint in the two-dimensional image. A value acquired by multiplying,by a correction constant, L1+L21+L31+L41 or L1+L22+L32+L42, acquired byadding them up, is computed as the height pixel count (h). When bothvalues can be computed, a longer value is set as the height pixel count,for example. In other words, each bone has a longest length in an imagewhen being captured from the front, and is displayed to be short whenbeing tilted in a depth direction with respect to a camera. Therefore,it is conceivable that a longer bone has a higher possibility of beingcaptured from the front, and has a value closer to a true value. Thus, alonger value is preferably selected.

In an example in FIG. 25 , the bone B1, the bone B51 and the bone B52,the bone B61 and the bone B62, and the bone B71 and the bone B72 areeach detected without overlapping. L1+L21+L31+L41 and L1+L22+L32+L42that are a total of the bones are acquired, and, for example, a valueacquired by multiplying, by a correction constant, L1+L22+L32+L42 on aleft leg side having a greater length of the detected bones is set asthe height pixel count.

In an example in FIG. 26 , the bone B1, the bone B51 and the bone B52,the bone B61 and the bone B62, and the bone B71 and the bone B72 areeach detected, and the bone B61 and the bone B71 of a right leg, and thebone B62 and the bone B72 of a left leg overlap. L1+L21+L31+L41 andL1+L22+L32+L42 that are a total of the bones are acquired, and, forexample, a value acquired by multiplying, by a correction constant,L1+L21+L31+L41 on a right leg side having a greater length of thedetected bones is set as the height pixel count.

In an example in FIG. 27 , the bone B1, the bone B51 and the bone B52,the bone B61 and the bone B62, and the bone B71 and the bone B72 areeach detected, and the bone B61 and the bone B71 of the right leg andthe bone B62 and the bone B72 of the left leg overlap. L1+L21+L31+L41and L1+L22+L32+L42 that are a total of the bones are acquired, and, forexample, a value acquired by multiplying, by a correction constant,L1+L22+L32+L42 on the left leg side having a greater length of thedetected bones is set as the height pixel count.

In the specific example 1, since a height can be acquired by adding uplengths of bones from a head to a foot, a height pixel count can beacquired by a simple method. Further, since at least a skeleton from ahead to a foot may be able to be detected by a skeleton estimationtechnique using machine learning, a height pixel count can be accuratelyestimated even when the entire person is not necessarily captured in animage as in a squatting state and the like.

<Specific Example 2> In the specific example 2, a height pixel count isacquired by using a two-dimensional skeleton model indicating arelationship between a length of a bone included in a two-dimensionalskeleton structure and a length of a whole body of a person on atwo-dimensional image space.

FIG. 28 is a human model (two-dimensional skeleton model) 301 that isused in the specific example 2 and indicates a relationship between alength of each bone on the two-dimensional image space and a length of awhole body on the two-dimensional image space. As illustrated in FIG. 28, a relationship between a length of each bone of an average person anda length of a whole body (a proportion of a length of each bone to alength of a whole body) is associated with each bone of the human model301. For example, a length of the bone B1 of a head is the length of thewhole body×0.2 (20%), a length of the bone B41 of a right hand is thelength of the whole body×0.15 (15%), and a length of the bone B71 of theright leg is the length of the whole body×0.25 (25%). Information aboutsuch a human model 301 is stored in the database 110, and thus anaverage length of a whole body can be acquired from a length of eachbone. In addition to a human model of an average person, a human modelmay be prepared for each attribute of a person such as age, sex, andnationality. In this way, a length (height) of a whole body can beappropriately acquired according to an attribute of a person.

In the specific example 2, as illustrated in FIG. 21 , the heightcomputation unit 108 acquires a length of each bone (S221). The heightcomputation unit 108 acquires a length of all bones (length on thetwo-dimensional image space) in a detected skeleton structure. FIG. 29is an example of capturing an image of a person in a squatting statediagonally from rear right and detecting a skeleton structure. In thisexample, since a face and a left side surface of a person are notcaptured, a bone of a head and bones of a left arm and a left handcannot be detected. Thus, each length of bones B21, B22, B31, B41, B51,B52, B61, B62, B71, and B72 that are detected is acquired.

Subsequently, as illustrated in FIG. 21 , the height computation unit108 computes a height pixel count from a length of each bone, based on ahuman model (S222). The height computation unit 108 refers to the humanmodel 301 indicating a relationship between lengths of each bone and awhole body as in FIG. 28 , and acquires a height pixel count from thelength of each bone. For example, since a length of the bone B41 of theright hand is the length of the whole body×0.15, a height pixel countbased on the bone B41 is acquired from the length of the bone B41/0.15.Further, since a length of the bone B71 of the right leg is the lengthof the whole body×0.25, a height pixel count based on the bone B71 isacquired from the length of the bone B71/0.25.

The human model referred at this time is, for example, a human model ofan average person, but a human model may be selected according to anattribute of a person such as age, sex, and nationality. For example,when a face of a person is captured in a captured image, an attribute ofthe person is identified based on the face, and a human model associatedwith the identified attribute is referred. An attribute of a person canbe recognized from a feature of a face in an image by referring toinformation acquired by performing machine learning on a face for eachattribute. Further, when an attribute of a person cannot be identifiedfrom an image, a human model of an average person may be used.

Further, a height pixel count computed from a length of a bone may becorrected by a camera parameter. For example, when a camera is placed ina high position and performs capturing in such a way that a person islooked down, a horizontal length such as a bone of a shoulder width isnot affected by a dip of the camera in a two-dimensional skeletonstructure, but a vertical length such as a bone from a neck to a waistis reduced as a dip of the camera increases. Then, a height pixel countcomputed from the horizontal length such as a bone of a shoulder widthtends to be greater than an actual height pixel count. Thus, when acamera parameter is used, an angle at which a person is looked down bythe camera is clear, and thus a correction can be performed in such away as to acquire a two-dimensional skeleton structure captured from thefront by using information about the dip. In this way, a height pixelcount can be more accurately computed.

Subsequently, as illustrated in FIG. 21 , the height computation unit108 computes an optimum value of the height pixel count (S223). Theheight computation unit 108 computes an optimum value of the heightpixel count from the height pixel count acquired for each bone. Forexample, a histogram of a height pixel count acquired for each bone asillustrated in FIG. 30 is generated, and a great height pixel count isselected from among the height pixel counts. In other words, a longerheight pixel count is selected from among a plurality of height pixelcounts acquired based on a plurality of bones. For example, values intop 30% are regarded valid, and height pixel counts by the bones B71,B61, and B51 are selected in FIG. 30 . An average of the selected heightpixel counts may be acquired as an optimum value, or a greatest heightpixel count may be set as an optimum value. Since a height is acquiredfrom a length of a bone in a two-dimensional image, when the bone cannotbe captured from the front, i.e., when the bone tilted in the depthdirection as viewed from the camera is captured, a length of the bone isshorter than that captured from the front. Then, a value having agreater height pixel count has a higher possibility of being capturedfrom the front than a value having a smaller height pixel count and is amore plausible value, and thus a greater value is set as an optimumvalue.

In the specific example 2, since a height pixel count is acquired basedon a bone of a detected skeleton structure by using a human modelindicating a relationship between lengths of a bone and a whole body onthe two-dimensional image space, a height pixel count can be acquiredfrom some of bones even when not all skeletons from a head to a foot canbe acquired. Particularly, a height pixel count can be accuratelyestimated by adopting a greater value from among values acquired from aplurality of bones.

<Specific Example 3> In the specific example 3, a skeleton vector of awhole body is acquired by fitting a two-dimensional skeleton structureto a three-dimensional human model (three-dimensional skeleton model)and using a height pixel count of the fit three-dimensional human model.

In the specific example 3, as illustrated in FIG. 22 , the heightcomputation unit 108 first computes a camera parameter, based on animage captured by the camera 200 (S231). The height computation unit 108extracts an object whose length is clear in advance from a plurality ofimages captured by the camera 200, and acquires a camera parameter froma size (pixel count) of the extracted object. Note that, a cameraparameter may be acquired in advance, and the acquired camera parametermay be obtained as necessary.

Subsequently, the height computation unit 108 adjusts an arrangement anda height of a three-dimensional human model (S232). The heightcomputation unit 108 prepares, for a detected two-dimensional skeletonstructure, the three-dimensional human model for computing a heightpixel count, and arranges the three-dimensional human model in the sametwo-dimensional image, based on the camera parameter. Specifically, a“relative positional relationship between a camera and a person in areal world” is determined from the camera parameter and thetwo-dimensional skeleton structure. For example, on the basis that aposition of the camera has coordinates (0, 0, 0), coordinates (x, y, z)of a position in which a person stands (or sits) are determined. Then,by assuming an image captured when the three-dimensional human model isarranged in the same position (x, y, z) as that of the determinedperson, the two-dimensional skeleton structure and the three-dimensionalhuman model are superimposed.

FIG. 31 is an example of capturing an image of a squatting persondiagonally from front left and detecting a two-dimensional skeletonstructure 401. The two-dimensional skeleton structure 401 includestwo-dimensional coordinate information. Note that, all bones arepreferably detected, but some of bones may not be detected. Athree-dimensional human model 402 as in FIG. 32 is prepared for thetwo-dimensional skeleton structure 401. The three-dimensional humanmodel (three-dimensional skeleton model) 402 is a model of a skeletonincluding three-dimensional coordinate information and having the sameshape as that of the two-dimensional skeleton structure 401. Then, as inFIG. 33 , the prepared three-dimensional human model 402 is arranged andsuperimposed on the detected two-dimensional skeleton structure 401.Further, the three-dimensional human model 402 is superimposed on thetwo-dimensional skeleton structure 401, and a height of thethree-dimensional human model 402 is also adjusted to thetwo-dimensional skeleton structure 401.

Note that, the three-dimensional human model 402 prepared at this timemay be a model in a state close to a pose of the two-dimensionalskeleton structure 401 as in FIG. 33 , or may be a model in an uprightstate. For example, the three-dimensional human model 402 with anestimated pose may be generated by using a technique for estimating apose in a three-dimensional space from a two-dimensional image by usingmachine learning. A three-dimensional pose can be estimated from atwo-dimensional image by learning information about a joint in thetwo-dimensional image and information about a joint in athree-dimensional space.

Subsequently, as illustrated in FIG. 22 , the height computation unit108 fits the three-dimensional human model to a two-dimensional skeletonstructure (S233). As in FIG. 34 , the height computation unit 108deforms the three-dimensional human model 402 in such a way that posesof the three-dimensional human model 402 and the two-dimensionalskeleton structure 401 match in a state where the three-dimensionalhuman model 402 is superimposed on the two-dimensional skeletonstructure 401. In other words, a height, an orientation of a body, andan angle of a joint of the three-dimensional human model 402 areadjusted, and optimization is performed in such a way as to eliminate adifference from the two-dimensional skeleton structure 401. For example,by rotating a joint of the three-dimensional human model 402 in amovable range of a person and also rotating the entire three-dimensionalhuman model 402, the entire size is adjusted. Note that, fitting(application) between a three-dimensional human model and atwo-dimensional skeleton structure is performed on a two-dimensionalspace (two-dimensional coordinates). In other words, a three-dimensionalhuman model is mapped in the two-dimensional space, and thethree-dimensional human model is optimized for a two-dimensionalskeleton structure in consideration of a change of the deformedthree-dimensional human model in the two-dimensional space (image).

Subsequently, as illustrated in FIG. 22 , the height computation unit108 computes a height pixel count of the fit three-dimensional humanmodel (S234). As in FIG. 35 , when there is no difference between thethree-dimensional human model 402 and the two-dimensional skeletonstructure 401 and poses match, the height computation unit 108 acquiresa height pixel count of the three-dimensional human model 402 in thatstate. With the optimized three-dimensional human model 402 in anupright state, a length of a whole body on the two-dimensional space isacquired based on a camera parameter. For example, a height pixel countis computed from lengths (pixel counts) of bones from a head to a footwhen the three-dimensional human model 402 is upright. Similarly to thespecific example 1, the lengths of the bones from the head to the footof the three-dimensional human model 402 may be added up.

In the specific example 3, a height pixel count is acquired based on athree-dimensional human model by fitting the three-dimensional humanmodel to a two-dimensional skeleton structure, based on a cameraparameter, and thus the height pixel count can be accurately estimatedeven when all bones are not captured at the front, i.e., when an erroris great due to all bones being captured on a slant.

<Normalization Processing> As illustrated in FIG. 19 , the imageprocessing apparatus 100 performs the normalization processing (S202)after the height pixel count computation processing. In thenormalization processing, as illustrated in FIG. 23 , the feature valuecomputation unit 103 computes a keypoint height (S241). The featurevalue computation unit 103 computes a keypoint height (pixel count) ofall keypoints included in the detected skeleton structure. The keypointheight is a length (pixel count) in the height direction from a lowestend (for example, a keypoint of any foot) of the skeleton structure tothe keypoint. Herein, as one example, the keypoint height is acquiredfrom a Y coordinate of the keypoint in an image. Note that, as describedabove, the keypoint height may be acquired from a length along avertical projection axis based on a camera parameter. For example, inthe example in FIG. 24 , a height (yi) of a keypoint A2 of a neck is avalue acquired by subtracting a Y coordinate of a keypoint A81 of aright foot or a keypoint A82 of a left foot from a Y coordinate of thekeypoint A2.

Subsequently, the feature value computation unit 103 determines areference point for normalization (S242). The reference point is a pointbeing a reference for representing a relative height of a keypoint. Thereference point may be preset, or may be able to be selected by a user.The reference point is preferably at the center of the skeletonstructure or higher than the center (in an upper half of an image in theup-down direction), and, for example, coordinates of a keypoint of aneck are set as the reference point. Note that coordinates of a keypointof a head or another portion instead of a neck may be set as thereference point. Instead of a keypoint, any coordinates (for example,center coordinates in the skeleton structure, and the like) may be setas the reference point.

Subsequently, the feature value computation unit 103 normalizes thekeypoint height (yi) by the height pixel count (S243). The feature valuecomputation unit 103 normalizes each keypoint by using the keypointheight of each keypoint, the reference point, and the height pixelcount. Specifically, the feature value computation unit 103 normalizes,by the height pixel count, a relative height of a keypoint with respectto the reference point. Herein, as an example focusing only on theheight direction, only a Y coordinate is extracted, and normalization isperformed with the reference point as the keypoint of the neck.Specifically, with a Y coordinate of the reference point (keypoint ofthe neck) as (yc), a feature value (normalization value) is acquired byusing the following equation (1). Note that, when a vertical projectionaxis based on a camera parameter is used, (yi) and (yc) are converted tovalues in a direction along the vertical projection axis.

[Mathematical 1]

f _(i)=(y _(i) −y _(c))/h  (1)

For example, when 18 keypoints are present, 18 coordinates (x0, y0),(x1, y1), . . . and (x17, y17) of the keypoints are converted to18-dimensional feature values as follows by using the equation (1)described above.

[Mathematical 2]

$\begin{matrix}\begin{matrix}\begin{matrix}\begin{matrix}{f_{0} = {( {y_{0} - y_{c}} )/h}} \\{f_{1} = {( {y_{1} - y_{c}} )/h}}\end{matrix} \\ \vdots \end{matrix} \\{f_{17} = {( {y_{17} - y_{c}} )/h}}\end{matrix} & (2)\end{matrix}$

FIG. 36 illustrates an example of a feature value of each keypointacquired by the feature value computation unit 103. In this example,since the keypoint A2 of the neck is the reference point, a featurevalue of the keypoint A2 is 0.0 and a feature value of a keypoint A31 ofa right shoulder and a keypoint A32 of a left shoulder at the sameheight as the neck is also 0.0. A feature value of a keypoint A1 of ahead higher than the neck is −0.2. Feature values of a keypoint A51 of aright hand and a keypoint A52 of a left hand lower than the neck are0.4, and feature values of the keypoint A81 of the right foot and thekeypoint A82 of the left foot are 0.9. When the person raises the lefthand from this state, the left hand is higher than the reference pointas in FIG. 37 , and thus a feature value of the keypoint A52 of the lefthand is −0.4. Meanwhile, since normalization is performed by using onlya coordinate of the Y axis, as in FIG. 38 , a feature value does notchange as compared to FIG. 36 even when a width of the skeletonstructure changes. In other words, a feature value (normalization value)according to the present example embodiment indicates a feature of askeleton structure (keypoint) in the height direction (Y direction), andis not affected by a change of the skeleton structure in the horizontaldirection (X direction).

As described above, in the present example embodiment, a skeletonstructure of a person is detected from a two-dimensional image, and eachkeypoint of the skeleton structure is normalized by using a height pixelcount (upright height on a two-dimensional image space) acquired fromthe detected skeleton structure. Robustness when classification, asearch, and the like are performed can be improved by using thenormalized feature value. In other words, since a feature valueaccording to the present example embodiment is not affected by a changeof a person in the horizontal direction as described above, robustnesswith respect to a change in orientation of the person and a body shapeof the person is great.

Furthermore, the present example embodiment can be achieved by detectinga skeleton structure of a person by using a skeleton estimationtechnique such as Open Pose, and thus learning data that learn a poseand the like of a person do not need to be prepared. Further,classification and a search of a pose and the like of a person can beachieved by normalizing a keypoint of a skeleton structure and storingthe keypoint in advance in a database, and thus classification and asearch can also be performed on an unknown pose. Further, a clear andsimple feature value can be acquired by normalizing a keypoint of askeleton structure, and thus persuasion of a user for a processingresult is high unlike a black box algorithm as in machine learning.

While the example embodiments of the present invention have beendescribed with reference to the drawings, the example embodiments areonly exemplification of the present invention, and variousconfigurations other than the above-described example embodiments canalso be employed.

Further, the plurality of steps (pieces of processing) are described inorder in the plurality of flowcharts used in the above-describeddescription, but an execution order of steps performed in each of theexample embodiments is not limited to the described order. In each ofthe example embodiments, an order of illustrated steps may be changedwithin an extent that there is no harm in context.

Further, each of the example embodiments described above can be combinedwithin an extent that a content is not inconsistent.

A part or the whole of the above-described example embodiment may alsobe described in supplementary notes below, which is not limited thereto.

1. An image selection apparatus, including:

a query acquisition unit that acquires query pose information that isinformation to be a query and indicates a pose of a person;

a search information acquisition unit that acquires a plurality ofpieces of search pose information that are information generated foreach of a plurality of target images to be a search target and indicatea pose of a person included in the target image;

a selection unit that selects, from among the plurality of pieces ofsearch pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and

a display control unit that displays, on a display unit, the targetimage associated with each of the two or more pieces of the search poseinformation selected by the selection unit, and also sets a displayposition of the target image by using a degree of reliability of thesearch pose information.

2. The image selection apparatus according to supplementary note 1described above, wherein

the display control unit displays, on the display unit, a degree ofreliability of the search pose information together with the targetimage associated with the search pose information.

3. The image selection apparatus according to supplementary note 1 or 2described above, wherein

the display control unit determines a display position of the targetimage by further using the degree of similarity.

4. The image selection apparatus according to supplementary note 1 or 2described above, wherein,

in a case where a predetermined condition is satisfied, the displaycontrol unit uses the degree of similarity instead of the search poseinformation or together with the search pose information whendetermining a display position of the target image.

5. The image selection apparatus according to any one of supplementarynotes 1 to 4 described above, wherein,

when the query acquisition unit acquires information that determines aquery image being an image including a person, the query acquisitionunit acquires, as the query pose information, pose informationassociated with the query image.

6. The image selection apparatus according to any one of supplementarynotes 1 to 5 described above, wherein

the search pose information is set by using a position of each of aplurality of joints, and

the degree of reliability is computed by using a number of joints usedwhen the search pose information is set, and a degree of reliability ofthe joint.

7. An image selection method, including,

executing by a computer:

query acquisition processing of acquiring query pose information that isinformation to be a query and indicates a pose of a person;

search information acquiring processing of acquiring a plurality ofpieces of search pose information that are information generated foreach of a plurality of target images to be a search target and indicatea pose of a person included in the target image;

selection processing of selecting, from among the plurality of pieces ofsearch pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and

display control processing of displaying, on a display unit, the targetimage associated with each of the two or more pieces of the selectedsearch pose information, and also setting a display position of thetarget image by using a degree of reliability of the search poseinformation.

8. The image selection method according to supplementary note 7described above, further including,

by the computer,

in the display control processing, displaying, on the display unit, adegree of reliability of the search pose information together with thetarget image associated with the search pose information.

9. The image selection method according to supplementary note 7 or 8described above, further including,

by the computer,

in the display control, determining a display position of the targetimage by further using the degree of similarity.

10. The image selection method according to supplementary note 7 or 8described above, further including,

by the computer,

in a case where a predetermined condition is satisfied, in the displaycontrol, using the degree of similarity instead of the search poseinformation or together with the search pose information whendetermining a display position of the target image.

11. The image selection method according to any one of supplementarynotes 7 to 10 described above, further including,

by the computer,

in the query acquisition, when acquiring information that determines aquery image being an image including a person, acquiring, as the querypose information, pose information associated with the query image.

12. The image selection method according to any one of supplementarynotes 7 to 11 described above, wherein

the search pose information is set by using a position of each of aplurality of joints, and

the degree of reliability is computed by using a number of joints usedwhen the search pose information is set, and a degree of reliability ofthe joint.

13. A program causing a computer to include:

a query acquisition function of acquiring query pose information that isinformation to be a query and indicates a pose of a person;

a search information acquisition function of acquiring a plurality ofpieces of search pose information that are information generated foreach of a plurality of target images to be a search target and indicatea pose of a person included in the target image;

a selection function of selecting, from among the plurality of pieces ofsearch pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and

a display control function of displaying, on a display unit, the targetimage associated with each of the two or more pieces of the search poseinformation selected by the selection unit, and also setting a displayposition of the target image by using a degree of reliability of thesearch pose information.

14. The program according to supplementary note 13 described above,wherein

the display control function displays, on the display unit, a degree ofreliability of the search pose information together with the targetimage associated with the search pose information.

15. The program according to supplementary note 13 or 14 describedabove, wherein

the display control function determines a display position of the targetimage by further using the degree of similarity.

16. The program according to supplementary note 13 or 14 describedabove, wherein,

in a case where a predetermined condition is satisfied, the displaycontrol function uses the degree of similarity instead of the searchpose information or together with the search pose information whendetermining a display position of the target image.

17. The program according to any one of supplementary notes 13 to 16described above, wherein,

when the query acquisition function acquires information that determinesa query image being an image including a person, the query acquisitionfunction acquires, as the query pose information, pose informationassociated with the query image.

18. The program according to any one of supplementary notes 13 to 17described above, wherein

the search pose information is set by using a position of each of aplurality of joints, and

the degree of reliability is computed by using a number of joints usedwhen the search pose information is set, and a degree of reliability ofthe joint.

REFERENCE SIGNS LIST

-   1 Image processing system-   10 Image processing apparatus (image selection apparatus)-   11 Skeleton detection unit-   12 Feature value computation unit-   13 Recognition unit-   100 Image processing apparatus (image selection apparatus)-   101 Image acquisition unit-   102 Skeleton structure detection unit-   103 Feature value computation unit-   104 Classification unit-   105 Search unit-   106 Input unit-   107 Display unit-   108 Height computation unit-   110 Database-   200 Camera-   300, 301 Human model-   401 Two-dimensional skeleton structure-   610 Query acquisition unit-   620 Search information acquisition unit-   630 Selection unit-   640 Display control unit

What is claimed is:
 1. An image selection apparatus, comprising: atleast one memory storing instructions; and at least one processorconfigured to execute the instructions to perform operations comprising:acquiring query pose information that is information to be a query andindicates a pose of a person; acquiring a plurality of pieces of searchpose information that are information generated for each of a pluralityof target images to be a search target and indicate a pose of a personincluded in the target image; selecting, from among the plurality ofpieces of search pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and displaying, on a display, the target imageassociated with each of the two or more pieces of the selected searchpose information, and setting a display position of the target image byusing a degree of reliability of the search pose information.
 2. Theimage selection apparatus according to claim 1, wherein the operationscomprise displaying, on the display, a degree of reliability of thesearch pose information together with the target image associated withthe search pose information.
 3. The image selection apparatus accordingto claim 1, wherein the operations comprise determining a displayposition of the target image by further using the degree of similarity.4. The image selection apparatus according to claim 1, wherein theoperations comprise, in a case where a predetermined condition issatisfied, using the degree of similarity instead of the search poseinformation or together with the search pose information whendetermining a display position of the target image.
 5. The imageselection apparatus according to claim 1, wherein the operationscomprise, when acquiring information that determines a query image beingan image including a person, acquiring, as the query pose information,pose information associated with the query image.
 6. The image selectionapparatus according to claim 1, wherein the search pose information isset by using a position of each of a plurality of joints, and the degreeof reliability is computed by using a number of joints used when thesearch pose information is set, and a degree of reliability of thejoint.
 7. An image selection method, comprising, by a computer:acquiring query pose information that is information to be a query andindicates a pose of a person; acquiring a plurality of pieces of searchpose information that are information generated for each of a pluralityof target images to be a search target and indicate a pose of a personincluded in the target image; selecting, from among the plurality ofpieces of search pose information, two or more pieces of the search poseinformation whose degree of similarity to the query pose informationsatisfies a reference; and displaying, on a display, the target imageassociated with each of the two or more pieces of the selected searchpose information, and also setting a display position of the targetimage by using a degree of reliability of the search pose information.8. A non-transitory computer-readable medium storing a program causing acomputer to perform operations comprising: acquiring query poseinformation that is information to be a query and indicates a pose of aperson; acquiring a plurality of pieces of search pose information thatare information generated for each of a plurality of target images to bea search target and indicate a pose of a person included in the targetimage; selecting, from among the plurality of pieces of search poseinformation, two or more pieces of the search pose information whosedegree of similarity to the query pose information satisfies areference; and displaying, on a display, the target image associatedwith each of the two or more pieces of the selected search poseinformation, and also setting a display position of the target image byusing a degree of reliability of the search pose information.